Supporting user-perceived usability benchmarking through a developed quantitative metric

Most user-centered assessment activities for ensuring usability are principally focused on performing formative evaluations, enrolling users to complete different tasks and thus obtaining indicators such as effectiveness and efficiency. However, when considering broader scenarios, such as in User Experience (UX) assessments, user perceived satisfaction (or perceived usability) is even much more relevant. There are different methods for measuring user perception, however most of them are mainly qualitative and based on individual assessments, providing little specific support to carry out comparisons–i.e., benchmarking on user-perceived usability. In this paper, we propose a quantitative metric to achieve comparative evaluations of usability perception based on Reaction Cards, a popular method for obtaining the user's subjective satisfaction in UX assessments. The metric was developed through an empirical study. Additionally, it has been validated with usability experts. Besides, we provide a supporting tool based on the developed metric, featuring a framework to store historical evaluations in order to obtain charts and benchmark levels for comparing perceived usability from different artifacts such as software products, applications categories, services, mockups, prototypes and so on. Furthermore, an evaluation involving usability professionals was achieved, providing satisfactory results to answer research questions, thus demonstrating the suitability of the approach proposedThis work was partially supported by the Spanish Government[grant number TIN2014-52129-R]; and the Madrid Research Council [grant number S2013/ICE-2715

Herramienta de Soporte para la Evaluación Subjetiva de la Usabilidad Mediante el Método Microsoft Reaction Cards

Este Trabajo Fin de Grado consiste en el desarrollo de una métrica cuantitativa para la evaluación comparativa de la usabilidad percibida por los usuarios (usabilidad subjetiva) a través del método Reaction Cards, además de una herramienta de soporte para llevar a cabo dicha evaluación, con el objetivo de poder realizar y comparar distintas evaluaciones, añadiendo una valoración numérica a cada evaluación. Reaction Cards es un método para medir la satisfacción del usuario, y este es un aspecto fundamental a la hora de realizar un análisis de usabilidad, por lo que es necesario poder comparar evaluaciones y establecer umbrales que permitan determinar qué solución es mejor. El método de Reaction Cards consiste en solicitar al usuario que seleccione entre 118 tarjetas con adjetivos aquellas que representan su experiencia con la aplicación, ampliando luego el análisis a obtener comentarios de las cinco tarjetas más representativas. Para interpretar estos resultados, en estudios anteriores se han dividido las tarjetas entre positivas y negativas. En este trabajo, se ha encontrado que la mejor forma de obtener una valoración numérica de cada tarjeta ha sido realizar un estudio con 55 sujetos para que valorasen cada tarjeta entre 1 (muy negativo) y 5 (muy positivo). La media acotada de la opinión de todos los sujetos es la puntuación asociada a cada tarjeta, y finalmente la puntuación final de la evaluación se forma como la suma de puntuaciones de las tarjetas seleccionadas entre la puntuación máxima con ese número de tarjetas, multiplicado por 100. La herramienta desarrollada es una herramienta web que permite el registro e inicio de sesión de usuarios. Cada usuario puede crear sus propios estudios (de los que es propietario), indicando la categoría, el nombre de la aplicación, el sistema operativo y la versión del mismo. Los usuarios ven los estudios de los que son propietarios, pudiendo añadir nuevas evaluaciones o acceder al detalle de los mismos y ver las evaluaciones realizadas con los comentarios recogidos. La herramienta proporciona además un sistema que permite comparar distintas evaluaciones de forma visual, mediante gráficas y nubes de palabras. El desarrollo se ha realizado en HTML/CSS/JavaScript para la vista, PHP para el controlador, y SQL para el modelo. La herramienta muestra las ventanas en el navegador, siendo páginas dinámicas que se generan en PHP en el servidor web, accediendo a la información del modelo contenido en una base de datos MySQL. Sobre la herramienta, se han realizado pruebas de usabilidad, en las que han participado 10 sujetos y ha permitido ver que la herramienta cumple con requisitos de facilidad de uso y de aprendizaje. En conclusión, se ha realizado una herramienta que permite gestionar y realizar estudios de usabilidad, aportando una puntuación numérica a cada evaluación de manera que se pueden comparar más fácilmente dos soluciones distintas, todo ello bajo la premisa de la facilidad de uso.This Bachelor Thesis consists in the development of a quantitative metric for the comparative evaluation of the usability perceived by users (subjective usability) through the Reaction Cards method, as well as a support tool to carry out that evaluation, in order to be able to make and to compare different assessments, by adding a numerical rating to each assessment. Reaction Cards is a method to measure user’s satisfaction, and this is a main aspect to consider when you are making a usability analysis, so it’s necessary to compare assessments and determine thresholds to see which solution is better. Reaction Cards method consists in showing the user a set of 118 cards, and encourage him to pick the cards that represents more his experience with the application, obtaining more comments from the user asking him about the five most representative cards. Previous studies have divided the cards between positive and negative to interpret these findings. However, in this thesis we found that the best way to obtain a numerical rating for each card has been doing a small study with 55 subjects in which the subjects select a rating for each card from 1 (very negative) to 5 (very positive). The average opinion of all subjects is the rating associated to each card, and finally the final score of an assessment is the sum of the ratings of selected cards, divided by the maximum score with that number of cards, multiplied by 100. Developed tool is a web application that allows user registration and login. Each user can create his own studies, which he is the owner, introducing category, application name, operating system and version. Users can see the detail of the studies of their own, and they can add new assessments or look at the comments received in previous assessments. The tool also gives a system that allows comparing different assessments in a visual way, using plots and word clouds. Development has been realized in HTML/CSS/JavaScript for the view, PHP for the controller and SQL for the model. The tool shows the interface in an explorer, each interface is a dynamic page generated by PHP in the web server, using the data allocated in a MySQL database. For testing, a usability testing has been performed, realized by 10 subjects, and this test has allowed to check that the tool complies with the ease of use and learning requisites, and has to improve in perceived usefulness. In conclusion, the developed tool allows to manage and realize usability studies, giving a numerical rating to each assessment and allowing to compare different evaluations easily, with an ease of use